Paraphrasing for Automatic Evaluation
نویسندگان
چکیده
This paper studies the impact of paraphrases on the accuracy of automatic evaluation. Given a reference sentence and a machine-generated sentence, we seek to find a paraphrase of the reference sentence that is closer in wording to the machine output than the original reference. We apply our paraphrasing method in the context of machine translation evaluation. Our experiments show that the use of a paraphrased synthetic reference refines the accuracy of automatic evaluation. We also found a strong connection between the quality of automatic paraphrases as judged by humans and their contribution to automatic evaluation.
منابع مشابه
ParaMetric: An Automatic Evaluation Metric for Paraphrasing
We present ParaMetric, an automatic evaluation metric for data-driven approaches to paraphrasing. ParaMetric provides an objective measure of quality using a collection of multiple translations whose paraphrases have been manually annotated. ParaMetric calculates precision and recall scores by comparing the paraphrases discovered by automatic paraphrasing techniques against gold standard alignm...
متن کاملParaphrasing Rules for Automatic Evaluation of Translation into Japanese
Automatic evaluation of translation quality has proved to be useful when the target language is English. In this paper the evaluation of translation into Japanese is studied. An existing method based on n-gram similarity between translations and reference sentences is difficult to apply to the evaluation of Japanese because of the agglutinativeness and variation of semantically similar expressi...
متن کاملTargeted Paraphrasing on Deep Syntactic Layer for MT Evaluation
In this paper, we present a method of improving quality of machine translation (MT) evaluation of Czech sentences via targeted paraphrasing of reference sentences on a deep syntactic layer. For this purpose, we employ NLP framework Treex and extend it with modules for targeted paraphrasing and word order changes. Automatic scores computed using these paraphrased reference sentences show higher ...
متن کاملParaphrasing for Style
We present initial investigation into the task of paraphrasing language while targeting a particular writing style. The plays of William Shakespeare and their modern translations are used as a testbed for evaluating paraphrase systems targeting a specific style of writing. We show that even with a relatively small amount of parallel training data, it is possible to learn paraphrase models which...
متن کاملManual and Automatic Paraphrases for MT Evaluation
Paraphrasing of reference translations has been shown to improve the correlation with human judgements in automatic evaluation of machine translation (MT) outputs. In this work, we present a new dataset for evaluating English-Czech translation based on automatic paraphrases. We compare this dataset with an existing set of manually created paraphrases and find that even automatic paraphrases can...
متن کامل